HPCC-30583 Fix parquet column reads #17946
Conversation
jackdelv
changed the title
|
https://track.hpccsystems.com/browse/HPCC-30583 |
|
@jackdelv The fact that XPATH is now supported may be worth a documentation update. I suspect that this issue -- where a Parquet file contains a field name that does not cleanly map to an ECL attribute name -- will come up quite often, like it does with XML and JSON files. It would be worth calling out to the ECL programmer that using XPATH as usual is the correct workaround. |
plugins/parquet/README.md
Outdated
| @@ -32,7 +32,14 @@ The Parquet Plugin offers the following main functions: | |||
|
|
|||
| The Read function allows ECL programmers to create an ECL dataset from both regular and partitioned Parquet files. It leverages the Apache Arrow interface for Parquet to efficiently stream data from ECL to the plugin, ensuring optimized data transfer. | |||
|
|
|||
| In order to read a parquet file it is necessary to define the record structure of the file you intend to read with the names of the fields as stored in the parquet file and the type that you wish to read them as. It is possible for a parquet file to have field names that contain characters that are incompatible with the ECL field name definition. For example, ECL field names are case insensitive causing an issue when trying to read parquet fields with uppercase letters. To read field names of this type an XPATH can be passed as seen in the following example: | |||
There was a problem hiding this comment.
I think you should capitalize 'Parquet'.
|
@dcamper I have updated the documentation to make it clearer to the user how to read a parquet file. Updated the example code to show an XPATH example. |
| @@ -1337,7 +1336,8 @@ void ParquetRowBuilder::nextField(const RtlFieldInfo *field) | |||
| nextFromStruct(field); | |||
| return; | |||
| } | |||
| auto column = result_rows->find(field->name); | |||
| (*array_visitor) = std::make_shared<ParquetArrayVisitor>(); | |||
| auto column = result_rows->find(field->xpath ? field->xpath : field->name); | |||
There was a problem hiding this comment.
not new/future/question:
for records with many fields, this linear search by field name per column into an arrow::Array could be quite expensive by the looks of it.
I assume their positions in the array are invariant between calls to nextField for a given ParquetRowBuilder instance?
And are probably also invariant from row to row?
Can you prepopulate a map before ParquetRowStream::nextRow() is called with a table from RtlFieldInfo -> to position in the arrow::Array, to avoid these find's ?
There was a problem hiding this comment.
I am not sure what you mean by a table from RtlFieldInfo? The result_rows object is a prepopulated map with the field name of the column and an array containing the values for that column. You are correct in that the positions are invariant from row to row and between calls to nextField.
There was a problem hiding this comment.
I misread. I thought the code was performing a find in arrow::Array per field name.
I see now that it's a find in 'result_rows' which is an unordered_map, whose rhs is a arrow::Array.
|
@ghalliday squashed |
ghalliday
merged commit 7b7a725
into
hpcc-systems:candidate-9.4.x
Type of change:
Checklist:
Smoketest:
Testing: